Method and apparatus for processing screen using device

ABSTRACT

A method and an apparatus for processing a screen by using a device are provided. The method includes obtaining, at the second device, a display screen displayed on the first device and information related to the display screen according to a screen display request regarding the first device, determining, at the second device, an additional screen based on the display screen on the first device and the information related to the display screen, and displaying the additional screen near the display screen on the first device.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 14/873,761, filed on Oct. 2, 2015, which claimed priority under 35U.S.C § 119(a) of a Korean patent application number 10-2014-0139071,filed on Oct. 15, 2014, in the Korean Intellectual Property Office, thedisclosure of which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to methods and apparatuses for processinga screen linked between devices.

BACKGROUND

In general, mobile devices are computer devices that are small-sizedenough to be fit in a pocket and include a display screen having a touchinput function or a small-sized keyboard. However, the screen of such amobile device is often smaller than the sizes of contents providedthereto, and thus a window should be additionally displayed or screenscrolling should be performed. Head-mounted display (HMD) devices shouldbe understood as various image display devices designed for users toview a virtual image while wearing the HMD devices on their heads likeeyeglasses. The HMD devices are capable of providing users with not onlya simple display function but also with various convenient functionswhich are combinations of a display function with augmented realitytechnology, N-screen technology, etc. However, a user of an HMD deviceshould indirectly attempt to control a virtual image displayed onglasses of the HMD device, for example, by making a gesture toward thevirtual image, making a voice input, or touching the arms of theeyeglasses without physically touching the virtual image. Thus, thevirtual image is difficult to be precisely controlled.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least theabove-mentioned problems and/or disadvantages and to provide at leastthe advantages described below. Accordingly, an aspect of the presentdisclosure is to provide methods and apparatuses for processing anadditional screen linked to an actual screen by using a device.

In accordance with an aspect of the present disclosure, a method ofprocessing a screen on a first device, which is performed by a seconddevice, is provided. The method includes obtaining, at the seconddevice, a display screen displayed on the first device and informationrelated to the display screen according to a screen display requestregarding the first device, determining, at the second device, anadditional screen based on the display screen on the first device andthe information related to the display screen, and displaying theadditional screen near the display screen on the first device.

The information related to the display screen may include at least oneof shape information of the first device, a location of the additionalscreen to be displayed near the display screen on the first device, asize of the additional screen, content information related to theadditional screen, and state information of the first device capturedusing a camera.

The state information of the first device may include a location andangle of a screen of the first device.

The state information of the first device may be periodically updatedusing the camera.

The displaying of the additional screen may include capturing an imageof the first device, determining a location and direction of a screen ofthe first device based on the captured image of the first device, anddisplaying the additional screen on a predetermined location on thescreen of the first device, based on the location and direction of thescreen of the first device.

The determining of the location and direction of the screen of the firstdevice may include receiving shape information of the first device fromthe first device, extracting feature points from the shape informationof the first device, and comparing feature points of the image of thefirst device with the feature points of the shape information of thefirst device to determine the location and direction of the screen ofthe first device.

The method may further include controlling the display screen on thefirst device and the additional screen to be linked to each otheraccording to user inputs performed on the first and second devices.

The controlling of the display screen on the first device and theadditional screen to be linked to each other may include controlling thedisplay screen on the first device and the additional screen on thesecond device to be linked to each other according to the user inputsperformed on the respective first and second devices.

The controlling of the display screen on the first device and theadditional screen to be linked to each other may include recognizing acommand to end the additional screen according to a user input receivedfrom the first device, and ending the additional screen by transmittingthe command to end the additional screen to the second device.

The controlling of the display screen on the first device and theadditional screen to be linked to each other may include recognizing auser input on the additional screen on the second device by using acamera, and ending the additional screen and transmitting informationindicating the end of the additional screen to the first device when theuser input is recognized.

In accordance with another aspect of the present disclosure, a method ofprocessing a screen by using a first device comprising an object, whichis performed by a second device, is provided. The method includesrecognizing, at the second device, the object by receiving informationrelated to the object from the first device, and transmittinginformation indicating a result of recognizing the object to the firstdevice, obtaining, at the second device, information regarding a displayscreen related to the recognized object according to an additionalscreen display request regarding the first device, determining, at thesecond device, an additional screen to be displayed on a predeterminedlocation on the object, based on the information regarding the displayscreen related to the object, and displaying the additional screen onthe predetermined location on the object.

The method may further include obtaining information regarding thedisplay region which is set on a predetermined location on the objectfrom the first device.

The information regarding the display screen related to the object mayinclude shape information of the object, a location and direction of theadditional screen to be displayed on the object, a size of theadditional screen, content information related to the additional screen,and state information of the object captured using a camera.

The state information of the object may include a location and angle ofthe object.

The state information of the object may be periodically updated usingthe camera.

The displaying of the additional screen may include capturing an imageof the object by using a camera, extracting information regarding alocation and angle of the object from the captured image of the object,and determining a location and direction in which the additional screenis to be displayed, based on the location and angle of the object.

The extracting of the information regarding the location and angle ofthe object may include receiving shape information of the object fromthe first device, extracting feature points from the shape informationof the object, and comparing the feature points of the shape informationof the object with feature points of the captured image of the object toextract the information regarding the location and angle of the object.

In accordance with yet another aspect of the present disclosure, amethod of processing an additional screen by using a first device, whichis performed by a second device, is provided. The method includesobtaining, at the second device, information regarding the additionalscreen and state information of the first device according to a screendisplay request regarding the first device, determining, at the seconddevice, a location on which the additional screen is to be displayed,based on the information regarding the additional screen and the stateinformation of the first device, displaying the additional screen on thedetermined location, and controlling the first and second devices to belinked to each other according to a user input related to the additionalscreen.

In accordance with still another aspect of the present disclosure, amethod of processing a screen on a second device, which is performed bya first device, is provided. The method includes transmitting, at thefirst device, a display screen and information related to the displayscreen to the second device according to a screen display request,receiving, at the first device, control information regarding anadditional screen displayed on the second device from the second device,and updating, at the first device, the information related to thedisplay screen according to the control information.

The information related to the display screen may include a location ofthe additional screen to be displayed near the display screen on thedevice, a size of the additional screen, and content information relatedto the additional screen.

In accordance with yet still another aspect of the present disclosure,an apparatus for processing a screen by using a device is provided. Theapparatus includes a display unit configured to display an image, acommunication unit configured to receive a display screen on the deviceand information related to the display screen according to a screendisplay request regarding the device, and a control unit configured toobtain an additional screen based on the information related to thedisplay screen obtained by the communication unit, display theadditional screen on the display unit such that the additional screen isdisplayed near the display screen on the device, and control the displayscreen on the device and the additional screen to be linked to eachother according to a user input.

The control unit may photograph a shape of the device by using a camera,extract information regarding a location and angle of a screen of thedevice from state information based on the shape of the device, anddetermine a region in which the additional screen is to be displayed,based on the location and angle of the screen of the device.

The control unit may control the display screen on the device and theadditional screen to be linked to each other according to a user inputperformed on the device and a user input related to the additionalscreen.

The control unit may recognize a user input on the additional screenthrough the camera, ends the additional screen when the user inputinstructs to end the additional screen, and transmits informationinforming the end of the additional screen to the device.

In accordance with still yet another aspect of the present disclosure, anon-transitory computer-readable recording medium is provided. Therecording medium has recorded thereon at least one program includinginstructions for performing a method of processing a screen on a firstdevice, performed by a second device. The method includes obtaining, atthe second device, a display screen displayed on the first device andinformation related to the display screen according to a screen displayrequest regarding the first device, determining, at the second device,an additional screen based on the information related to the displayscreen, and displaying the additional screen near the display screen onthe first device.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a block diagram of is a block diagram of a system configuredto process a screen between a first device and a second device accordingto an embodiment of the present disclosure;

FIG. 2 is a block diagram of a system configured to process a screenbetween a first device and a second device according to an embodiment ofthe present disclosure;

FIG. 3 is a block diagram of a system configured to process a screenbetween a first device and a second device according to an embodiment ofthe present disclosure;

FIG. 4 is a block diagram of a system configured to process a screenbetween a first device and a second device according to an embodiment ofthe present disclosure;

FIG. 5 is a diagram illustrating a case in which a second deviceprocesses an additional screen to be linked to a screen on a firstdevice according to an embodiment of the present disclosure;

FIG. 6 is a diagram illustrating a case in which a second devicedisplays a virtual image type additional screen around an objectaccording to an embodiment of the present disclosure;

FIG. 7 is a diagram illustrating a case in which a second devicedisplays additional screens to be linked to an object according to anembodiment of the present disclosure;

FIG. 8 is a flowchart of a method of processing a screen linked to ascreen displayed on a first device, performed by a second device,according to an embodiment of the present disclosure;

FIG. 9 is a flowchart of a method of processing a screen linked to ascreen displayed on a first device, performed by a second device,according to an embodiment of the present disclosure;

FIG. 10 is a flowchart of a method of processing a screen displayed on asecond device, performed by a first device, according to an embodimentof the present disclosure;

FIG. 11 is a flowchart of a method of processing a screen by using firstdevice on which no screen is displayed, performed by a second device,according to an embodiment of the present disclosure;

FIG. 12 is a flowchart of a method of processing a screen by using afirst device that additionally includes an object, performed by a seconddevice, according to an embodiment of the present disclosure;

FIG. 13 is a flowchart of a method of displaying an additional screen tobe linked to a screen displayed on a first device, performed by a seconddevice, according to an embodiment of the present disclosure;

FIG. 14 is a flowchart of a method of displaying a screen to be linkedto a screen displayed on a first device including an object, performedby a second device, according to an embodiment of the presentdisclosure;

FIG. 15 is a flowchart of a method of controlling an additional screen,performed by a second device, according to an embodiment of the presentdisclosure;

FIG. 16 is a flowchart of a method of controlling an additional screento be ended, performed by a first device, according to an embodiment ofthe present disclosure;

FIG. 17 is a flowchart of a method of controlling an additional screento be displayed on a first device, performed by a second device,according to an embodiment of the present disclosure;

FIGS. 18A and 18B are diagrams illustrating a method of displaying avirtual image type additional screen near a display screen on a firstdevice and controlling the additional screen, performed by a seconddevice, according to various embodiments of the present disclosure;

FIGS. 19A and 19B are diagrams illustrating a method of displaying avirtual image type additional screen near a display screen on a firstdevice and controlling the additional screen, performed by a seconddevice, according to various embodiments of the present disclosure;

FIG. 20 is a diagram illustrating a method of displaying athree-dimensional (3D) image type additional screen around a displayscreen on a first device and controlling the additional screen,performed by a second device, according to an embodiment of the presentdisclosure;

FIG. 21 is a block diagram of a first device according to an embodimentof the present disclosure;

FIG. 22 illustrates the exterior of a second device according to anembodiment of the present disclosure; and

FIGS. 23 and 24 are block diagrams of second devices according tovarious embodiments of the present disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purpose only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

It will be understood that, although the terms ‘first’, ‘second’,‘third’, etc., may be used herein to describe various elements,components, regions, layers and/or sections, these elements, components,regions, layers and/or sections should not be limited by these terms.These terms are only used to distinguish one element, component, region,layer or section from another region, layer or section.

The specific terms used in the present disclosure are not intended torestrict the scope of the present disclosure and only used for a betterunderstanding of (to facilitate the understanding of) variousembodiments of the present disclosure. In the present disclosure,general terms that have been widely used nowadays are selected, ifpossible, in consideration of functions of the present disclosure, butnon-general terms may be selected according to the intentions oftechnicians in the this art, precedents, or new technologies, etc. Someterms may be arbitrarily chosen by the present applicant. In this case,the meanings of these terms will be explained in corresponding parts ofthe present disclosure in detail. Thus, the terms used herein should bedefined not based on the names thereof but based on the meanings thereofand the whole context of the present disclosure.

It will be further understood that the terms ‘comprise’ and/or‘comprising,’ when used in this specification, specify the presence ofstated features, integers, operations, elements, and/or components, butdo not preclude the presence or addition of one or more other features,integers, operations, elements, components, and/or groups thereof. Asused herein, the term ‘and/or’ includes any and all combinations of oneor more of the associated listed items. Expressions such as ‘at leastone of,’ when preceding a list of elements, modify the entire list ofelements and do not modify the individual elements of the list.

FIG. 1 is a block diagram of a system configured to process screeninformation between a first device and a second device according to anembodiment of the present disclosure.

Referring to FIG. 1, the first device 1000 may request the second device2000 to display an additional screen, and process the additional screenaccording to a control command and screen information received from thesecond device 2000. Here, the additional screen may be a virtual imagelinked to a main screen.

Examples of the first device 1000 may include, but are not limited to, awearable terminal, a smart phone, a personal computer (PC), a tablet PC,a mobile phone, a smart television (TV), a personal digital assistant(PDA), a laptop computer, a media player, a global positioning system(GPS) device, an electronic book terminal, a digital broadcastingterminal, a navigation device, a kiosk, a Motion Picture Experts Groupphase 1 or phase 2 (MPEG-1 or MPEG-2) audio layer 3 (MP3) player, adigital camera, and other mobile or non-mobile computing devices. Thefirst device 1000 may include various devices capable of receiving atouch input, e.g., an electronic bulletin board, a touch table, etc. Thefirst device 1000 may include a display or may not include a display.

Also, the first device 1000 may have a communication function and a dataprocessing function but is not limited thereto. The first device 1000includes various devices capable of receiving information from thesecond device 2000 via a network and executing the information.

The second device 2000 may receive information regarding an additionalscreen together with a screen display request from the first device1000. The screen display request may be generated by the first device1000 or the second device 2000 according to a user input. The seconddevice 2000 may capture an image of the first device 1000 through acamera, and analyze a state of the first device 1000 based on the image.The second device 2000 may determine a location and a direction in whichthe additional screen is to be displayed, based on state information ofthe first device 1000 and the information regarding the additionalscreen. The second device 2000 may display the additional screen to belinked to a screen on the first device 1000. The second device 2000 maycontrol the additional screen to be linked to a screen on the firstdevice 1000 according to a user input. In this case, the user input forthe second device 2000 may be a touch input or a button input which isinput by a user but is not limited thereto.

According to an embodiment of the present disclosure, the second device2000 may capture an image of the shape of the first device 1000 by usinga camera and transmit the image of the shape to the first device 1000.The first device 1000 determines a region or a location in which theadditional screen is to be displayed, based on the image of the shape ofthe first device 1000, and transmit to the second device 2000information regarding the region or the location in which the additionalscreen is to be displayed.

Examples of the second device 2000 may include, but are not limited to,a head-mounted display (HMD), a PC, a smart TV, a digital broadcastingterminal, and other mobile/non-mobile computing devices. The seconddevice 2000 may include various devices capable of receiving a touchinput, e.g., an electronic bulletin board, a touch table, etc.

The second device 2000 may display the additional screen on a displaythereof in a form of augmented reality (AR), mixed reality (MR), orvirtual reality (VR).

The second device 2000 may display a virtual screen on a transparent ora non-transparent display.

The second device 2000 may be eyeglasses or a device worn on the bodysuch as a hand band having a communication function and a dataprocessing function, but is not limited thereto.

The first device 1000 and the second device 2000 may be connected via anetwork. The network may be a wire network such as a local area network(LAN), a wide area network (WAN) or a value-added network (VAN), orvarious types of wireless network such as a mobile radio communicationnetwork or a satellite communication network.

As described above, according to various embodiments of the presentdisclosure, various visual information may be provided by exposing, onthe screen of the second device, information related to an actual screendisplayed on the first device 1000 using a user input for the seconddevice 2000 while a user views the first device 1000 using the seconddevice 2000, and a convenient interactive environment may be provided byalternately controlling a virtual screen and the actual screen.

FIG. 2 is a block diagram of a system configured to process a screenbetween a first device and a second device according to an embodiment ofthe present disclosure.

The first device 1000 may additionally include an object 1000 a havinglimited functions or having no function. According to an embodiment ofthe present disclosure, the object 1000 a may be either a band having atleast a control function and a sensor function, or a wrist band or asticker having no function.

For example, a user may wear the object 1000 a, such as a band havinglimited functions, on his or her wrist, and the first device 1000 may bea separate device that may be out of sight or a device or an applicationwhich is present in a network.

According to an embodiment of the present disclosure, if the firstdevice 1000 sets, as a display region, a human wrist or a natural objectsuch as white paper of a diary or wallpaper, the second device 2000informs the first device 1000 of a natural object recognition event, anddisplays an additional screen on a location corresponding to the naturalobject according to a request for the additional screen, which isreceived from the first device 1000, when the second device 2000recognizes the natural object using a camera.

According to an embodiment of the present disclosure, the first device1000 may designate an object, based on a captured image. The seconddevice 2000 may receive captured image information from the first device1000, and determine a location on which an additional screen is to bedisplayed. For example, when the first device 1000 captures an image ofa user's wrist and designates the image of the user's wrist as anobject, the second device 2000 displays a virtual watch on the image ofthe user's wrist. When the first device 1000 captures an image of ablank note and designates the image of the blank note as an object, thesecond device 2000 may display a virtual tablet PC on the image of theblank image.

FIG. 3 is a block diagram of a system configured to process a screenbetween a first device and a second device according to an embodiment ofthe present disclosure.

Referring to FIG. 3, the first device 1000 may be a device or anapplication which is present in a cloud 1000 b. The first device 1000present in the cloud 1000 b may additionally include an object 1000 a.

FIG. 4 is a block diagram of a system configured to process a screenbetween a first device and a second device according to an embodiment ofthe present disclosure.

Referring to FIG. 4, the first device 1000 may be a control device or anapplication which is present in the second device 2000. The first device1000 present in the second device 2000 may additionally include anobject 1000 a.

FIG. 5 is a diagram illustrating a case in which a second device 2000processes an additional screen to be linked to a screen of a firstdevice 1000 according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the first device1000 may be a mobile device and the second device 2000 may be aneyeglasses type HMD.

According to an embodiment of the present disclosure, the eyeglass typeHMD may include a lens via which an actual space is viewed, and adisplay unit attached to a location on the lens and configured todisplay a virtual image. The eyeglass type HMD will be described in moredetail with reference to FIG. 22 below.

FIG. 5 illustrates a glass screen that is a combination of an actualscreen that comes into a user's view via an eyeglass type HMD and avirtual image.

The glass screen 100 means a screen that a user views via the lens andthe display unit of the HMD.

Referring to FIG. 5, a user may view a display screen 40 that isactually displayed on the first device 1000, and additional screens 50,60, 70, and 80 displayed on the second device 2000 by using the HMD.Here, the display screen 40 displayed on the first device 1000 is anactual screen viewed via the lens of the HMD, and the additional screens50, 60, 70, and 80 displayed on the second device 2000 are virtualimages viewed near the first device 1000 via the display unit of theHMD.

That is, on the glass screen 100 of the second device 200, the displayscreen 40 displayed on the first device 1000 is displayed and theadditional screens 50, 60, 70, and 80 are displayed near the displayscreen 40.

Thus, a user would feel as if the display screen 40 displayed on thefirst device 1000 and the additional screens 50, 60, 70, and 80 weredisplayed on the second device 2000.

According to an embodiment of the present disclosure, when the seconddevice 2000 displays additional screens via the first device 1000 onwhich nothing is displayed, the second device 2000 may display a virtualwatch image on a surface of the first device 1000, and display virtualapplication folder icons at the left and right sides of the virtualwatch image.

FIG. 6 is a diagram illustrating a case in which a second devicedisplays a virtual image type additional screen around an objectaccording to an embodiment of the present disclosure.

Referring to FIG. 6, a first device may additionally include an object1000 a having limited functions or having no function. The first device1000 may transmit object-related information to the second device 2000.

The second device 2000 obtains predetermined object-related informationfrom the first device 1000. The second device 2000 recognizes the object1000 a using a camera 2110 based on the object-related information, andinforms the first device 1000 of information containing a result ofrecognizing the object 1000 a. The second device 2000 obtains displayinformation related to the object 1000 a according to a screen displayrequest received from the first device 1000. The second device 2000determines an additional screen to be displayed on a predeterminedlocation on a captured image of the object 1000 a based on the displayinformation. The second device 2000 may display the additional screen ona location corresponding to the image of the object 1000 a.

FIG. 7 is a diagram illustrating a case in which a second devicedisplays additional screens to be linked to an object according to anembodiment of the present disclosure.

Referring to FIG. 7, a plurality of additional screens 52, 62, 72, and82 are displayed on a glass screen 100 on the second device 2000 to belinked to an object 42 that is in the form of an actual watch. Accordingto an embodiment of the present disclosure, the additional screens 52,62, 72, and 82 may be virtual images displayed on a screen of an HMD.Here, the object 42 that is in the form of an actual watch is an imageof an actual watch viewed via a lens of the HMD, and the additionalscreens 52, 62, 72, and 82 are virtual images viewed near the image ofthe actual watch via a display unit of the HMD.

A user would feel as if the additional screens 52, 62, 72, and 82 weredisplayed near the object 42 of the first device 1000 via the seconddevice 2000.

FIG. 8 is a flowchart of a method of processing screen informationlinked to a screen displayed on a first device, performed by a seconddevice, according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 8, in operation 210, the second device 2000obtains a display screen displayed on the first device 1000 andinformation related to the display screen according to a screen displayrequest regarding the first device 1000. For example, the informationrelated to the display screen may include information regarding alocation of additional screens to be displayed near the display screenon the first device 1000, the sizes of the additional screens, thecontents of the additional screens, and state information of the firstdevice 1000 captured using a camera. According to an embodiment of thepresent disclosure, the additional screens may be virtual imagesdisplayed on a screen of an HMD.

In this case, the screen display request regarding the first device 1000may be generated by the first device 1000 or the second device 2000according to a user input. According to an embodiment of the presentdisclosure, the screen display request regarding the first device 1000may be transmitted by connecting the first device 1000 and the seconddevice 2000 via a network or recognizing the first device 1000 using acamera of the second device 2000 but is not limited thereto.

The first device 1000 may select a menu and generate a command todisplay additional screens related to the menu according to a user inputsuch as a gesture, a touch, or voice. The second device 2000 may receivethe command to display additional screens related to the menu from thefirst device 1000. Also, the second device 2000 may receive additionalscreens related to a menu that are set beforehand by the first device1000. The additional screens may be, for example, a plurality of virtualimages (such as “PHOTO”, “APP”, S-health“, and “Gallery”) which are notdisplayed on a screen of the first device 1000 due to space restraintsof the first device 1000. Examples of the additional screens may includevarious types of screens that may be displayed on the screen of thefirst device 1000.

Also, the second device 2000 captures an image of a shape of the firstdevice 1000 using a camera. The second device 2000 may photograph thefirst device 1000 to determine the location and angle of the screen onthe first device 1000 by using a camera, and receive, in real time, animage of the shape of the first device 1000 captured by the camera.

In operation 220, the second device 2000 determines additional screensto be displayed near the display screen on the first device 1000, basedon the information related to the display screen on the first device1000.

In operation 230, the second device 2000 displays the additional screensnear the display screen on the first device 1000. According to anembodiment of the present disclosure, the second device 2000 may displaythe additional screens on predetermined locations near the displayscreen on the first device 1000. The second device 2000 may determinelocations and directions in which the additional screens are to bedisplayed, based on captured shape information of the first device 1000and the information related to the display screen on the first device1000. In a method of determining locations and directions in whichadditional screens are to be displayed according to an embodiment of thepresent disclosure, the second device 2000 may recognize the shape ofthe first device 1000 photographed by a camera, extract informationregarding the location and angle of a screen of the first device 1000from information regarding the shape of the first device 1000, anddetermine locations and directions corresponding to regions in which theadditional screens are to be displayed, based on the location and angleof the screen of the first device 1000. According to an embodiment ofthe present disclosure, the second device 2000 may photograph thedisplay screen on the first device 1000 by using the camera, and displaythe display screen and the determined additional screens.

Accordingly, according to an embodiment of the present disclosure, auser may view, via the second device 2000, the display screen displayedon the first device 1000 and the additional screens displayed on thesecond device 2000.

FIG. 9 is a flowchart of a method of processing a screen linked to ascreen displayed on a first device, performed by a second device,according to an embodiment of the present disclosure.

In FIG. 9, operations 215, 225, and 235 are substantially the same asoperations 210, 220, and 230 of FIG. 4 and are not thus described hereagain.

Referring to FIGS. 1 and 9, in operation 245, the second device 2000controls additional screens to be linked to a display screen displayedon the first device 1000. The additional screens on the second device2000 and the display screen on the first device 1000 are controlled tobe linked to one another. A user may change a screen of the first device1000 by manipulating the additional screens on the second device 2000 orchange the additional screens on the second device 2000 by selecting amenu of the first device 1000. The second device 2000 may control theadditional screens thereof to be linked to the display screen on thefirst device 1000 according to a user input that is input for the firstdevice 1000 or that is input by performing a camera recognition gestureon the second device 2000. In this case, the user input for the firstdevice 1000 may be input by performing a specific gesture, a touchinput, or a voice input by a user, but is not limited thereto. The userinput for the second device 2000 may be input by performing a camerarecognition gesture, a touch input, a button input, or a voice input,but is not limited thereto. For example, when a ‘Gallery’ menu on anadditional screen displayed to be linked to the display screen on thefirst device 1000 is touched by performing a specific gesture, the‘Gallery’ menu may be displayed on the display screen on the firstdevice 1000. Also, the first device 1000 may end the additional screenby transmitting a command to end the additional screen to the seconddevice 2000. The second device 2000 may end the additional screen when auser input is recognized on the additional screen via a camera, andtransmit information indicating the end of the additional screen to thefirst device 1000. When the first device 1000 receives the informationindicating the end of the additional screen from the second device 2000,the first device 1000 may display a result of processing a result ofprocessing the additional screen on an actual screen. According to anembodiment of the present disclosure, when the second device 2000 is anHMD, examples of a gesture performed on the additional screen mayinclude a touch gesture performed by touching the additional screen, anda motion gesture performed, for example, by drawing an ‘X’ on theadditional screen.

FIG. 10 is a flowchart of a method of processing a screen displayed on asecond device, performed by a first device, according to an embodimentof the present disclosure.

Referring to FIGS. 1 and 10, in operation 250, the first device 1000transmits a display screen and information related to the display screento the second device 2000 according to a user input requesting a screento be displayed. The information related to the display screen mayinclude a location and direction in which an additional screen is to beexposed near the display screen on the first device 1000, the size ofthe additional screen, and contents information related to theadditional screen. According to an embodiment of the present disclosure,the additional screen may be a virtual image displayed on a screen of anHMD.

In operation 260, the first device 1000 receives user input informationregarding the additional screen from the second device 2000. Forexample, the first device 1000 receives information for selecting theadditional screen from the second device 2000.

In operation 270, the first device 1000 updates the information relatedto the display screen of the first device 1000, based on the user inputinformation received from the second device 2000. For example, the firstdevice 1000 may update the display screen linked to the additionalscreen selected by the second device 2000.

FIG. 11 is a flowchart of a method of processing a screen by using afirst device on which no screen is displayed, performed by a seconddevice, according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 11, in operation 310, the second device 2000receives information regarding an additional screen according to anadditional screen display request regarding the first device 1000. Forexample, the information regarding the additional screen may includelocation information, content information, and size information of theadditional screen to be exposed on a surface of the first device 1000.

In operation 320, the second device 2000 obtains an image of the shapeof the first device 1000 by using a camera. The second device 2000 maydetect the location and angle of the first device 1000 by photographingthe first device 1000 by using the camera, receive in real time an imageof the shape of the first device 1000 captured using the camera, andperiodically update the image of the shape of the first device 1000 byusing the camera.

In operation 330, the second device 2000 determines a location on whichthe additional screen is to be displayed, based on the captured image ofthe shape of the first device 1000 and the information regarding theadditional screen. The second device 2000 extracts the coordinates ofthe location of the first device 1000 and the angle of a screen of thefirst device 1000 by analyzing the shape of the first device 1000, anddetermines a specific location on the first device 1000 on which theadditional screen is to be displayed, based on the coordinates of thelocation and the angle.

In operation 340, the second device 2000 displays the additional screenon the determined location. For example, the second device 2000 maydisplay a watch screen on the first device 1000, and may displayapplication folder icons at left and right sides of the watch screen.

In operation 350, the second device 2000 controls the additional screento be linked to the first device 1000. The second device 2000 maycontrol the additional screen to be linked to the first device 1000according to a user input for the first device 1000 and a user input forthe second device 2000. In this case, the user input for the firstdevice 1000 may be input by performing a specific gesture, a touchinput, a button input, or a voice input by a user, but is not limitedthereto. The user input for the second device 2000 may be input byperforming a camera recognition gesture, a touch input, a button input,or a voice input, but is not limited thereto. For example, when thefirst device 1000 receives touch information for a virtual applicationfolder icon from the second device 2000, the first device 1000 transmitsinformation regarding an additional screen related to the touchedapplication folder to the second device 2000. Also, the second device2000 displays, on a predetermined surface of the first device 1000, anadditional screen related to an application folder received from thefirst device 1000.

FIG. 12 is a flowchart of a method of processing a screen, by using afirst device which additionally includes an object, performed by asecond device, according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 12, in operation 365, the second device 2000obtains predetermined object-related information from the first device1000. For example, when an object is a band, the object-relatedinformation may be information regarding the shape or location of a bandrelated to the first device 1000.

In operation 370, the second device 2000 recognizes the object using acamera, based on the object-related information, and transmitsinformation indicating a result of recognizing the object to the firstdevice 1000.

In operation 375, the second device 2000 obtains display informationrelated to the object according to a screen display request, which isreceived from the first device 1000.

For example, the display information related to the object may includethe location of an additional screen to be displayed on the object, thesize of the additional screen, content information related to theadditional screen, and shape information of the object captured using acamera.

In operation 380, the second device 2000 determines an additional screento be displayed on a predetermined location on the object, based on thedisplay information related to the object.

In operation 385, the second device 2000 displays the additional screenon a location corresponding to the object. According to an embodiment ofthe present disclosure, the second device 2000 may display theadditional screen on the object 1000 a related to the first device 1000.The second device 2000 may determine a location on which the additionalscreen is to be displayed, based on captured state information of theobject 1000 a. In a method of determining a location on which anadditional screen is to be displayed according to an embodiment of thepresent disclosure, the second device 2000 may recognize stateinformation of the object 1000 a captured using a camera, extractsinformation regarding the coordinates of the location of and the angleof the object 1000 a from the state information of the object 1000 a,and determine a region on which the additional screen is to bedisplayed, based on the location and angle of the object 1000 a.

In a method of processing a screen by using a first device according toan embodiment of the present disclosure, an object may be a naturalobject such as a human body part, white paper of a diary, and acalendar.

That is, the first device 1000 sets a human palm as a display region andinforms the second device 2000 of information regarding the displayregion. The second device 2000 informs the first device 1000 of arecognition event when the human palm is recognized using a camera, andmay display a virtual keyboard or a calendar in a region correspondingto the human palm according to a request for an additional screen fromthe first device 1000.

FIG. 13 is a flowchart of a method of displaying an additional screen tobe linked to a screen displayed on a first device, performed by a seconddevice, according to an embodiment of the present disclosure.

Referring to FIGS. 1 and 13, according to an embodiment of the presentdisclosure, the first device 1000 is a mobile device and the seconddevice 2000 is an HMD.

In operation 410, the first device 1000 checks whether an additionalscreen display request related to a display screen thereof is receivedthrough a user input performed using a gesture, a touch, or voice iswhile the display screen is displayed on the first device 1000.According to an embodiment of the present disclosure, the additionalscreen may be a virtual image displayed on a screen of an HMD.

In operation 415, when it is determined that the additional screendisplay request is received, the first device 1000 transmits, to thesecond device 2000, information regarding the additional screen relatedto the display screen of the first device 1000 together with atriggering signal corresponding to the additional screen displayrequest. For example, when a user wants to expand and view the displayscreen of the first device 1000, the first device 1000 may transmit theinformation regarding the additional screen to the second device 2000while requesting the second device 2000 to display the additionalscreen.

In this case, the information regarding the additional screen mayinclude shape information of the first device 1000, content informationrelated to the additional screen, location information of the additionalinformation to be displayed on the display screen of the first device1000, image information related to the additional screen, and the sizeinformation of the additional screen. For example, when the first device1000 is an e-book, the content information related to the additionalscreen may be a virtual table of contents of the e-books, the locationinformation of the additional screen may be a right side of a screen ofthe e-book, and the size information of the additional screen may besubstantially the same as the size information of a display screen onthe e-book. Also, when a user selects the virtual table of contents ofthe e-book on the additional screen that is in the form of a virtualimage, the location information of the additional screen may be a rightside of a screen of the list of the virtual e-books on the e-book, andthe size information of the additional screen may be 70% of the size ofthe screen of the virtual table of contents of the e-book.

In operation 420, the second device 2000 receives the additional screendisplay request and the information regarding the additional screen fromthe first device 1000.

In operation 425, when the second device 2000 receives informationregarding an additional screen and the additional screen display requestfrom the first device 1000, the second device 2000 photographs the shapeof the first device 1000 by using a camera.

In operation 430, the second device 2000 recognizes the state of thefirst device 100 based on the shape of the first device 100 so as todetermine a location on which the additional screen is to be displayed.In this case, information regarding the state of the recognized firstdevice 1000 may include device type and the location and angle of thedisplay screen.

In operation 435, the second device 2000 extracts information regardingthe location and angle of a screen of the first device 1000 from theinformation regarding the state of the first device 1000, and determinesa location and direction in which the additional screen is to bedisplayed on the first device 1000, based on the location and angle ofthe screen of the first device 1000 and the information regarding theadditional screen. Information regarding the location and angle of thescreen of the first device 1000 may be extracted using methods accordingto various embodiments of the present disclosure.

According to an embodiment of the present disclosure, the second device2000 receives shape information of the first device 1000 from the firstdevice 1000. The second device 2000 extracts feature points from theshape information of the first device. The second device 2000 determinesthe location of the displays screen on the first device 1000 bycomparing feature points of an image of the first device 1000 capturedusing a camera with the feature points of the shape information of thefirst device. Also, the second device 2000 determines the angle of thedisplay screen, based on the relationship between the feature points ofthe image of the first device 1000 and the feature points of the shapeinformation. For example, the second device 2000 determines that theangle of the display screen on the first device 1000 exactly faces afront direction when the distances between four feature points at thefour vertices of the first device 1000 are the same, and determines thatthe angle of the display screen is tilted when the distances betweenfour feature points at the four vertices of the first device 1000 arenot the same.

In operation 440, the second device 2000 displays the additional screenlinked to the display screen on the first device 1000, based on thelocation of the display screen and information related to the additionalscreen. According to an embodiment of the present disclosure, thedisplay screen on the first device 1000 and the additional screen aredisplayed on a glass screen of an HMD. Thus, a user may view the displayscreen on the first device 1000 and the additional screen via the seconddevice 2000.

FIG. 14 is a flowchart of a method of displaying a screen to be linkedto a screen displayed on a first device including an object, performedby a second device, according to an embodiment of the presentdisclosure.

Referring to FIGS. 1 and 14, according to an embodiment of the presentdisclosure, the first device 1000 is a mobile device and the seconddevice 2000 is an HMD. The first device 1000 additionally includes anobject 1000 a having limited functions or having no function asillustrated in FIG. 2. For example, the object 1000 a may be a wristband, a sticker, or a calendar. Otherwise, the object 1000 a may beprovided separately from the first device 1000.

In operation 450, the first device 1000 transmits information related tothe object 1000 a. The information related to the object 1000 a may beshape information of or location information of the object 1000 a. Forexample, when the object 1000 a is a band, the information related tothe object 1000 a may be shape information or location information ofthe band.

In operation 455, when receiving the information related to the object1000 a from the first device 1000, the second device 2000 recognizes theshape of the object 1000 a using a camera, based on the informationrelated to the object 1000 a. According to an embodiment of the presentdisclosure, when the object 1000 a is a wrist band having no function,the first device 1000 cannot recognize a user's viewing of the wristband and thus the second device 2000 may recognize the wrist band usingthe camera. However, when the object 1000 a is a wrist band havinglimited functions, the first device 1000 is capable of recognizing auser's viewing of the wrist band and may thus directly request thesecond device 2000 to display an additional screen without performing arecognition operation. According to an embodiment of the presentdisclosure, when the first device 1000 sets a natural object, e.g., ahuman wrist, a diary, a calendar, etc., as a display region, the seconddevice 2000 may recognize the natural object using the camera.

In operation 460, after the object 1000 a is recognized using thecamera, the second device 2000 transmits information indicating theresult of recognizing the object to the first device 1000. According toan embodiment of the present disclosure, when a wrist band is recognizedusing the camera, the second device 2000 may inform the first device1000 of a wrist band recognition event.

In operation 465, the first device 1000 checks an additional screendisplay request is received through a user input such as a user'smotion, a touch, or voice while a screen is displayed on the firstdevice 1000.

In operation 470, when it is determined that the additional screendisplay request is received, the first device 1000 transmits informationregarding an additional screen thereof to the second device 200,together with a triggering signal corresponding to this request. Forexample, when a user wants to display a menu screen on the object 1000a, the first device 1000 may transmit a signal requesting to display theadditional screen and the information regarding the additional screen tothe second device 2000.

In this case, the information regarding the additional screen mayinclude the contents of the additional screen, location information ofthe additional screen to be displayed on a predetermined location on theobject 1000 a, content information related to the additional screen, andthe size of the additional screen. According to an embodiment of thepresent disclosure, the additional screen may be a virtual imagedisplayed on a screen of an HMD.

In operation 475, the second device 2000 obtains the additional screendisplay request and the information regarding the additional screen fromthe first device 1000.

In operation 480, when receiving the additional screen display requestand the information regarding the additional screen regarding the firstdevice 1000, the second device 2000 photographs the shape of the object1000 a using the camera.

In operation 485, the second device 2000 recognizes the state of theobject 1000 a based on the shape of the object 1000 a so as to determinea location on which the additional screen is to be displayed. In thiscase, the recognized state of the object 1000 a may include the locationand angle of the object 1000 a. For example, the second device 2000 mayrecognize the location and angle of a wrist band which is the object1000 a.

According to an embodiment of the present disclosure, the second device2000 receives shape information of the object 1000 a from the object1000 a. The second device 2000 extracts feature points from the shapeinformation. The second device 2000 determines the location of thescreen on the first device 1000 by comparing feature points of an imageof the object 1000 a captured using a camera with the feature points ofthe shape information of the object 1000 a. Also, the second device 2000determines the angle of the screen, based on the relationship betweenthe feature points of the image of the object 1000 a and the featurepoints of the shape information. For example, the second device 2000determines that the angle of the object 1000 a exactly faces a frontdirection when the distances between four feature points at the fourvertices of the object 1000 a are the same, and determines that theangle of the object 1000 a is tilted when the distances between fourfeature points at the four vertices of the object 1000 a are not thesame.

In operation 490, the second device 2000 extracts the coordinates of thelocation of the object 1000 a and the angle of the object 1000 a fromthe state information of the object 1000 a, and determines a locationand direction in which the additional screen is to be displayed, basedon received information related to the additional screen and thelocation and angle of the object 1000 a.

In operation 495, the second device 2000 displays the additional screenin the determined location and direction of the object 1000 a, based onthe determined location of the object 1000 a and the information relatedto the additional screen.

FIG. 15 is a flowchart of a method of controlling an additional screen,performed by a second device, according to an embodiment of the presentdisclosure.

Referring to FIGS. 1 and 15, according to an embodiment of the presentdisclosure, the first device 1000 is a mobile device and the seconddevice 2000 is an HMD. The additional screen may be a virtual imagedisplayed on a screen of an HMD.

In operation 510, the first device 1000 checks whether a first userinput is received through a gesture, a touch, or voice.

In operation 515, when the first user input is received, the firstdevice 1000 transmits a control command to the second device 2000according to the first user input.

In operation 525, the second device 2000 generates and processes anevent related to displaying of an additional screen, which correspondsto the first user input.

In operation 530, the second device 2000 displays the additional screenafter the event related to displaying of an additional screen isprocessed.

In operation 520, the second device 2000 checks whether a second userinput such as a virtual touch gesture or voice is performed on theadditional screen.

In operation 515, the second device 2000 transmits a control command tothe first device 1000 according to the second user input.

In operation 535, the first device 1000 generates and processes an eventrelated to displaying of an additional screen, which corresponds to thesecond user input.

In operation 540, the first device 1000 updates the display screen afterthe event related to displaying of an additional screen is processed.For example, when the second device 2000 selects an additional screen ofa specific menu by virtually touching the additional screen, the firstdevice 1000 update and display the selected additional screen of thespecific menu.

FIG. 16 is a flowchart of a method of controlling an additional screenon a second device to be ended, performed by a first device, accordingto an embodiment of the present disclosure.

Referring to FIGS. 1 and 16, according to an embodiment of the presentdisclosure, the first device 1000 is a mobile device and the seconddevice 2000 is an HMD.

In operation 545, the first device 1000 checks whether a command to endan additional screen is received through a user input. In this case, theuser input for the first device 1000 may be a specific gesture input, atouch input, a button input, or a voice input, but is not limitedthereto.

In operation 550, the first device 1000 transmits a command to end theadditional screen to the second device 2000.

In operation 555, the second device 2000 ends the additional screenaccording to the command to end the additional screen, which is receivedfrom the first device 1000.

FIG. 17 is a flowchart of a method of controlling a display screen on afirst device, performed by a second device, according to an embodimentof the present disclosure.

Referring to FIGS. 1 and 17, according to an embodiment of the presentdisclosure, the first device 1000 is a mobile device and the seconddevice 2000 is an HMD.

In operation 560, the second device 2000 checks, by using a camera,whether a command to end an additional screen is input through a virtualgesture or voice. According to an embodiment of the present disclosure,the command to end the additional screen may be input by selecting amenu on the additional screen or the additional screen by performing atouch gesture, selecting a region outside the additional screen, orperforming a specific gesture, e.g., drawing an ‘X’.

In operation 565, when the command to end the additional screen isreceived, the second device 2000 ends the additional screen.

In operation 570, the second device 2000 transmits informationindicating the end of the additional screen to the first device 1000. Inthis case, the information indicating the end of the additional screenmay include at least one among a video signal, an audio signal, and ahaptic signal.

In operation 575, the first device 1000 displays a screen thereonaccording to the information indicating the end of the additionalscreen, which is received from the second device 2000.

FIGS. 18A and 18B are diagrams illustrating a method of displayingvirtual image type additional screens around a display screen on a firstdevice and controlling the additional screens, performed by a seconddevice, according to various embodiments of the present disclosure.

Referring to FIGS. 1, 18A, and 18B, according to an embodiment of thepresent disclosure, the first device 1000 is a wrist type terminal andthe second device 2000 is an HMD device.

Referring to FIG. 18A, an additional screen including at least one menuimage is displayed on the second device 2000 to be linked to a screendisplayed on the first device 1000. Reference numeral 600 denotes aglass screen of the second device 2000.

When the second device 2000 recognizes, by using a camera, a virtualtouch gesture or voice that is input on a ‘Gallery’ menu 610corresponding to the additional screen, the second device 2000 transmitsa user input corresponding to selecting of the additional screen to thefirst device 1000.

Referring to FIG. 18B, the first device 1000 may display a ‘Gallery’menu 625 corresponding to the selected additional screen according tothe user input received from the second device 2000.

FIGS. 19A and 19B are diagrams illustrating a method of displaying avirtual image type additional screen near a display screen on a firstdevice and controlling the additional screens, performed by a seconddevice, according to various embodiments of the present disclosure.

Referring to FIGS. 1, 19A, and 19B, according to an embodiment of thepresent disclosure, a first device 710 is an e-book terminal and thesecond device 2000 is an HMD device.

Referring to FIG. 19A, an additional screen including at least onevirtual image is displayed on the second device 2000 to be linked to adisplay screen on an e-book terminal 710. A reference numeral 700denotes a glass screen region of the second device 2000. Here, thedisplay screen of the e-book terminal 710 is an actual screen viewed viaa lens of the HMD device, and additional screens 715 and 720 displayedon the second device 2000 are virtual images viewed near the displayscreen on the e-book terminal 710 via a display unit of the HMD device.

When the second device 2000 recognizes, by using a camera, a user inputperformed on the additional screen 715 which is a table of contentsthrough a virtual touch gesture or voice, the second device 2000displays an additional screen 720 corresponding to a preview image. Whenthe additional screen 720 which is the preview image is selected througha virtual touch gesture or voice, the second device 2000 transmits aselection signal corresponding to the selected additional screen to thee-book terminal 710.

Referring to FIG. 19B, the e-book terminal 710 may display a displayscreen 725 corresponding to the preview image thereon according to auser input received from the second device 2000.

FIG. 20 is a diagram illustrating a method of displayingthree-dimensional (3D) image type additional screens near a displayscreen on a first device and controlling the additional screens,performed by a second device, performed by a second device according toan embodiment of the present disclosure.

Referring to FIGS. 1 and 20, according to an embodiment of the presentdisclosure, a first device 810 is a tablet terminal and the seconddevice 2000 is an HMD device.

Referring to FIG. 20, the second device 2000 displays a 3D AR screen atvarious angles to be linked to a display screen displayed on a tabletterminal 810. A reference numeral 800 denotes a glass screen on thesecond device 2000.

The second device 2000 recognizes a motion 820 of a user's finger on anAR additional screen (e.g., a human-body screen) by using a camera. Thesecond device 2000 moves the location of an image 830 of a cross-sectionof a human body displayed on the tablet terminal 810 according to themotion 820. The user of the second device 2000 may select a portion ofthe image 830 corresponding to the cross-section of the human body whilemoving the location of his/her finger.

When an additional screen corresponding to the cross-section of thehuman body is selected by moving the user's finger, the second device2000 transmits a selection signal to the tablet terminal 810.

Accordingly, as illustrated in FIG. 20, the tablet terminal 810 maydisplay an image corresponding to the cross-section of the human body tobe selected according to the selection signal received from the seconddevice 2000.

FIG. 21 is a block diagram of a first device according to an embodimentof the present disclosure.

Referring to FIG. 21, the first device 1000 may include a user inputunit 1100, a display unit 1200, a control unit 1300, a sensor unit 1400,a communication unit 1500, an audio/video (A/V) input unit 1600, amemory 1700, and a notification module 1800.

The user input unit 1100 means a unit configured for a user to inputdata for controlling the first device 1000. For example, examples of theuser input unit 1100 may include, but are not limited to, a key pad, adome switch, a touch pad (a touch-type capacitive touch pad, apressure-type resistive overlay touch pad, an infrared sensor type touchpad, a surface acoustic wave conduction touch pad, an integration typetension measurement touch pad, a piezo effect type touch pad, etc.), ajog wheel, a jog switch, etc.

The user input unit 1100 may receive a user input instructing to displayan additional screen from the second device 2000 of FIG. 1.

The display unit 1200 displays information processed by the first device1000. For example, the display unit 1200 may display a user interfacesuch as a menu.

In general, the control unit 1300 controls overall operations of thefirst device 1000. For example, the control unit 1300 may generallycontrol the user input unit 1100, the display unit 1200, the sensor unit1400, the communication unit 1500, and the A/V input unit 1600 byexecuting programs stored in the memory 1700.

In more detail, the control unit 1300 may control the communication unit1500 to control displaying of the additional screen through a user inputsuch as a gesture, a touch, or voice. The first device 1000 may select amenu according to a user input and generate a command to display anadditional screen related to the menu. The control unit 1300 may requestthe second device 2000 to display the additional screen, and process ascreen according to a control command corresponding to a user inputreceived from the second device 2000. The control unit 1300 may providethe second device 2000 with an additional screen display request andinformation related to the additional screen. For example, theinformation related to additional screen may include a location on whichthe additional screen is to be displayed, contents related to theadditional screen, and the size of the additional screen. Also, thecontrol unit 1300 may update information related to a display screen,based on control information regarding the additional screen receivedfrom the second device 20000.

The sensor unit 1400 may sense the state of the first device 1000 or thestate of the vicinity of the first device 1000, and transmit informationcontaining a result of sensing the state of the first device 1000 or thestate of the vicinity of the first device 1000 to the control unit 1300.

The sensor unit 1400 may include at least one among a geomagneticsensor, an acceleration sensor, a temperature/humidity sensor 1430, aninfrared sensor 1440, a gyroscope sensor, a position sensor (e.g., aGPS), a proximity sensor, and a red, green, and blue (RGB) sensor (anilluminance sensor).

The communication unit 1500 may include at least one element configuredto establish communication between the first device 1000 and the seconddevice 2000.

Also, the communication unit 1500 may control the second device 2000 todisplay the additional screen, and exchange information for changing anoperation of the additional screen with the second device 2000.

The A/V input unit 1600 is configured to input an audio signal or avideo signal, and may include a camera 1610, a microphone 1620, etc. Thecamera 1610 may obtain image frames such as still images or videos viaan image sensor in a video call mode or a shooting mode. An imagecaptured by the image sensor may be processed using the control unit1300 or an additional image processor (not shown).

An image frame processed by the camera 1610 may be stored in the memory1700 or transmitted to the outside via the communication unit 1500. Themicrophone 1620 receives an external sound signal and converts it intoelectrical voice data. For example, the microphone 1620 may receive asound signal from an external device or a speaker. The microphone 1620may employ various noise removal algorithms for removing noise generatedwhen an external sound signal is input to the microphone 1620.

The memory 1700 may store a program for processing and controloperations of the control unit 1300, and store data input to or outputfrom the first device 1000 (e.g., location information of an additionalscreen to be linked to the display screen on the first device 1000,content information related to the additional screen, size informationof the additional screen, etc.).

The notification module 1800 may generate a signal informing an eventrelated to the additional screen of the first device 1000. Examples ofthe event related to the additional screen generated by the first device1000 may include receiving a call signal, receiving a message, inputtinga key signal, informing a schedule, etc. The notification module 1800may output this signal in the form of a video signal via the displayunit 1200.

FIG. 22 illustrates the exterior of a second device according to anembodiment of the present disclosure.

FIG. 22 illustrates a case in which the second device 2000 is aneyeglass type HMD device.

Although a frame 2010 of the HMD device illustrated in FIG. 22 is anall-in-one type, the shape and structure of a frame of a HMD deviceintroduced in the present disclosure is not limited to those of theframe illustrated in FIG. 22.

For example, referring to FIG. 22, the frame 2010 of the HMD device mayinclude a connecting member (not shown) and thus at least a portion ofthe frame 2010 may be foldable. For example, an HMD device according toan embodiment of the present disclosure includes the foldable frame 2010and may be thus folded in storage when the HMD device is not in use by auser, thereby minimizing a space occupied by the HMD device.

An HMD device according to an embodiment of the present disclosure mayfurther include an elastic band (not shown) allowing the HMD device tobe fixedly put on a user's head, regardless of the size of the user'shead.

According to an embodiment of the present disclosure, the frame 2010 maybe configured such that a lens 2005 is attachable to or detachable fromthe frame 2010. According to an embodiment of the present disclosure, aHMD device may not include the lens 2005. Also, the lens 2005illustrated in FIG. 22 is integrally formed with a nose bridge butvarious embodiments of the present disclosure are not limited to thatshown in FIG. 22. For example, a nose bridge of an HMD device accordingto an embodiment of the present disclosure may be integrally formed withthe frame 2010.

The lens 2005 may be formed of a transparent material so that a user mayview an actual space via the lens 2005. The lens 2005 may be formed of amaterial that passes light, which forms an image to be displayed adisplay unit 2030, there through. Examples that may be used to form thelens 2010 include plastic such as polycarbonate or glass but are notlimited thereto.

The control unit 2600 may be connected to the HMD device in a wired orwireless manner Although FIG. 22 illustrates that the control unit 2600is located at a left side of the frame 2010, various embodiments of thepresent disclosure, are not limited thereto. For example, the controlunit 2600 may be located at a right side of the frame 2010 or located ona front surface of the HMD device to be adjacent to a camera 2050.

The control unit 2600 may receive data from, for example, the camera2050 or user input units 2040, analyze the received data, and generateinformation to be transmitted to a user of the HMD device via at leastone of the display unit 2030 and an audio output unit 2020. Theinformation to be transmitted to the user may include at least one amongan image, a text, a video, and audio but is not limited thereto.

The camera 2050 may be included in the display unit 2030 or may belocated on the frame 2010, separately from the display unit 2030. Thecamera 2050 may be a camera that is used in smart phones or asmall-sized camera such as webcams. For example, the camera 2050 may bemounted on a location determined beforehand to be appropriate forobtaining an image by making a gesture using a user's hand. For example,as illustrated in FIG. 22, the camera 2050 may be mounted on a locationadjacent to a user's eyes when the user wears the HMD device and be thuscapable of capturing an image to be substantially the same as thatrecognized through the user's eyes.

The user input unit 2040 may include, but is not limited to, at leastone of a touch pad that may be operated by a user's finger and a buttonthat may be operated through a push operation. Although FIG. 22illustrates that the user input units 2040 are disposed at sides of theframe 2010, the user input units 2040 may be located on another locationon the HMD device.

The user input units 2040 are configured to receive a user input. Theuser input may include data or a signal that is input by a user togenerate an event causing the HMD device to start or end a predeterminedoperation.

For example, the user input units 2040 may include an ‘on’/‘off’ switchto power on or off the HMD device. Also, the user input unit 2040according to an embodiment of the present disclosure may receive a userinput for controlling an image displayed using the HMD device.

The display unit 2030 may be located on a left upper end of the lens2005 and configured using a semi-transparent optical waveguide (e.g., aprism) as illustrated in FIG. 22. The display unit 2030 illustrated inFIG. 22 may reflect light output from a projector built in the HMDdevice to focus an image on the foveae of the retinas of the eyes of auser who wears the HMD device. However, the display unit 2030 includedin the HMD device according to an embodiment of the present disclosureis not limited to that shown in FIG. 22, and may be configured accordingto various methods and in various structures to display an image near auser's eyes. For example, according to an embodiment of the presentdisclosure, the display unit 2030 may be a transparent display or anon-transparent display.

Although not shown in FIG. 22, according to an embodiment of the presentdisclosure, the HMD device may further include a microphone (not shown).The microphone may receive a user's voice, sound in an ambientenvironment of the HMD device, etc.

The sound output unit 2020 may be configured in the form of an earphoneto be mounted on the ears of a user of the HMD device. The sound outputunit 2020 may be fixedly mounted on the HMD device as illustrated inFIG. 22 but various embodiments of the present disclosure are notlimited thereto. For example, the sound output unit 2020 may beconfigured to be attachable to or detachable from the HMD device, sothat a user of the HMD device may selectively put the sound output unit2020 on his or her ears. For example, the sound output unit 2020 may bea bone conduction speaker.

A power supply unit 2015 is illustrated as being provided on an endportion of the frame 2010 of the HMD device. However, variousembodiments of the present disclosure are not limited thereto and thepower supply unit 2015 may be disposed on various locations on the frame2010 of the HMD device. The power supply unit 2015 supplies power foroperating the HMD device to the elements of the HMD device. Although notshown, the power supply unit 2015 may include a battery for charging,and a cable or a cable port via which power may be received from theoutside.

FIGS. 23 and 24 are block diagrams of second devices according tovarious embodiments of the present disclosure. FIGS. 23 and 24illustrate cases in which the second devices are eyeglass-type HMDdevices.

Referring to FIG. 23, the HMD device 2000 according to an embodiment ofthe present disclosure, includes at least a camera 2050, a user inputunit 2200, a display unit 2030, a communication unit 2500, and a controlunit 2600.

However, all of the elements illustrated in FIG. 23 are notindispensable elements of the HMD device 2000. The HMD device mayfurther include other elements or may include only some of the elementsillustrated in FIG. 23.

For example, referring to FIG. 24, the HMD device 2000 according to anembodiment of the present disclosure may further include a microphone2055, an output unit 2300, a memory 2400, a sensor unit 2700, and aninterface unit 2800, in addition to the camera 2050, the user input unit2200, the display unit 2030, the communication unit 2500, and thecontrol unit 2600.

The camera 2050 is configured to photograph an object in an actualspace. An image of the object captured by the camera 2050 may be amoving picture image or continuous still images. The HMD device may be,for example, an eyeglass-type device having a communication function anda data processing function. The camera 2050 facing the front of a userwhen the user wears the HMD device may photograph an object in an actualspace.

According to an embodiment of the present disclosure, the camera 2050 isconfigured to photograph the first device 1000 of FIG. 1, an objectrelated to the first device 1000, or a user's gesture. For example, thecamera 2050 may obtain information regarding a part of a user's body orinformation regarding the user's gesture. The information regarding thepart of the user's body may include an image of the part of the user'sbody. The information regarding the user's gesture may include an imageobtained by photographing a body part of the user who makes the gesture.

Two or more cameras 2050 may be provided according to an embodiment ofthe second device 2000.

The microphone 2055 receives an external sound signal and converts itinto electrical voice data in a call mode, a recording mode, a voicerecognition mode, etc. In the call mode, the electrical voice data maybe converted and output into a form that may be transmitted to a mobilecommunication base station via a mobile communication module 2510. Themicrophone 2055 may employ various noise removing algorithms to removenoise generated when an external sound signal is input to the microphone2055.

The user input unit 2200 receives a user input for controlling the HMDdevice 2000. The user input unit 2200 may receive a touch input and akey input performed on the HMD device 2000. Also, the user input unit2200 may receive a user's gesture photographed by the camera 2050 fromthe camera 2050.

The display unit 2030 displays information regarding an additionalscreen of a virtual image processed by the control unit 2600 of the HMDdevice 2000 to a user.

According to an embodiment of the present disclosure, the display unit2030 may display an additional screen of a virtual image on apredetermined location on a display screen displayed on the first device1000.

Also, according to an embodiment of the present disclosure, the displayunit 2030 may display an additional screen of a virtual image on apredetermined location on an object related to the first device 1000.

Also, the display unit 2030 may display a user interface for selectingan additional screen of a virtual image processed by the control unit1600.

The display unit 2030 may include (not shown) at least one among a touchscreen, electronic paper, a liquid crystal display (LCD), a thin-filmtransistor-LCD (TFT-LCD), a light-emitting diode (LED), an organic LED(OLED), a flexible display, a 3D display, and a transparent display.

The output unit 2300 is configured to output an audio signal, a videosignal, or an alarm signal. The output unit 2300 may include an audiooutput unit 2020, an alarm unit 2330, a haptic module 2340, etc.

The audio output unit 2020 outputs audio data received from thecommunication unit 2500 or control unit 2600 or stored in the memory2400 in a call signal receiving mode, a call mode, a recording mode, avoice recognition mode, a broadcast receiving mode, etc. Also, the soundoutput unit 2020 outputs an audio signal related to a function performedby the HMD device 2000, e.g., a call-signal receiving sound, a messagereceiving sound, etc. The audio output unit 2020 may include a speaker,a buzzer, etc.

The alarm unit 2330 outputs a signal informing an additionalscreen-related event of the HMD device 2000. Examples of the additionalscreen-related event generate by the HMD device 2000 include receivingof a call signal, receiving a message, inputting of a key signal, etc.The alarm unit 2330 outputs a signal informing the additionalscreen-related event in a form different from an audio signal or a videosignal. When a call signal or a message is received, the alarm unit 2330may output a signal informing this fact. Also, when a key signal isinput, the alarm unit 2330 may output a signal as a feedback of theinputting of the key signal. A user may recognize that the additionalscreen-related event occurs by checking the signal output from the alarmunit 2330. Alternatively, in the HMD device 2000, the signal informingthe occurrence of the additional screen-related event may be output viathe display unit 2030 or the sound output unit 2020.

The haptic module 2340 generates various haptic effects that a user mayfeel. A representative example of haptic effects generated by the hapticmodule 2340 may be a vibration effect. When the haptic module 2340generates vibration as a haptic effect, the intensity and pattern of thevibration generated by the haptic module 2340 may be converted anddifferent vibrations may be synthesized and output or may besequentially output.

In addition to vibration, the haptic module 2340 may generate varioushaptic effects, e.g., an effect using a stimulus caused by a verticalmovement of an arrangement of pins on a contact skin surface, an effectusing a stimulus caused by a jet force of air jetted from a nozzle or asuction force of air sucked through an air-inlet, an effect using astimulus caused when a skin surface is brushed, an effect using astimulus caused when an electrode is contacted, an effect using astimulus caused by an electrostatic force, an effect achieved when asense of cold or heat is generated using a material capable of suckingheat or generating heat, etc. The haptic module 2340 may be configuredsuch that a haptic effect is not only delivered when haptic module 2340is directly touched but also is felt through the sensation of a user'smuscle such as a finger or an arm.

The memory 2400 may store a program for processing and controloperations of the control unit 2600, and temporarily store data that isinput or output (e.g., message, still images, video, locationinformation of an additional screen to be linked to the display screenof the first device 1000, content information related to the additionalscreen, size information of the additional screen, etc.).

The memory 2400 may include at least one storage medium among a flashmemory type memory, a hard disk type memory, a multimedia card microtype memory, a card type memory, a random access memory (RAM), and aread-only memory (ROM). Also, the memory 2400 may include a web storagethat has a storage function and that is operated in the Internet.

The communication unit 2500 receives information related to anadditional screen from the first device 1000, and exchanges, with thefirst device 1000, information required for the HMD device 2000 todisplay the additional screen on the display unit 2030 and to change thedisplayed additional screen, based on the information related toadditional screen. According to an embodiment of the present disclosure,the communication unit 2500 may exchange required information withperipheral devices or a server.

The communication unit 2500 may include a mobile communication module2510, a near-field communication (NFC) module 2520, a broadcastreceiving module 2530, an Internet module 2540, a GPS module 2550, etc.

The mobile communication module 2510 exchanges a radio signal with atleast one among a base station, an external terminal, and a server in amobile communication network. Here, the radio signal may include a voicecall signal, a video call signal, or various types of data to be usedwhen text/multimedia messages are exchanged. Thus, the HMD device 2000may exchange a voice call signal, a video call signal, or atext/multimedia message with an external device.

The NFC module 2520 means a module configured to establish NFC. NFCtechnologies may include BLUETOOTH™, radio-frequency identification(RFID), infrared data association (IrDA), ultra wideband (UWB), ZIGBEE™,etc.

The broadcast casting module 2530 is configured to receive a broadcastsignal by using various broadcasting systems, and particularly, digitalmultimedia broadcasting-terrestrial (DMB-T), a digital multimediabroadcasting-satellite (DMB-S), media forward link only (MediaFLO),digital video broadcasting-handheld (DVB-H), integrated services digitalbroadcast-terrestrial (ISDB-T), etc. Also, the broadcast casting module2530 may be configured to be suitable for not only such digitalbroadcasting systems but also other various broadcasting systems capableof providing a broadcast signal.

The Internet module 2540 means a module configured to access a wirelessInternet and may be installed inside or outside the second device 2000.Wireless local area network (WLAN), wireless broadband (Wibro), worldinteroperability for microwave access (Wimax), high-speed downlinkpacket access (HSDPA), etc. may be used as wireless Internet technology.

The GPS module 2550 may receive location information from a plurality ofGPS satellites.

The sensor unit 2700 senses a current state of the HMD device 2000,e.g., the location of the HMD device 2000, whether the HMD device 2000is touched by a user, etc., and generates a sensing signal forcontrolling an operation of the HMD device 2000. For example, the sensorunit 2700 may perform a sensing function related to whether theinterface unit 2800 is coupled to an external device, etc.

The sensor unit 2700 may include a proximity sensor 2710, a motionsensor 2730, etc. The proximity sensor 2710 may detect whether an objectis approaching the HMD device 2000 or is present near the HMD device2000 without a physical contact. The proximity sensor 2710 may detect anobject that is approaching or is present near the HMD device 2000 bychecking a change in an alternating magnetic field or a static magneticfield, a rate of change in an electrostatic capacitance, etc.

The motion sensor 2730 senses the location or motion of the seconddevice 2000 by using an acceleration sensor, a gyro sensor, etc. Theacceleration sensor is a device capable of converting a change inacceleration in a direction into an electrical signal, and has beenwidely used as micro-electromechanical systems (MEMS) technology hasadvanced. The gyro sensor is a sensor that is configured to measure anangular speed and may sense a direction deviating from a referencedirection.

The interface unit 2800 acts as an interface with all of externaldevices connected to the HMD device 2000. Examples of external devicesconnected to the HMD device 2000 include a wire/wireless headset, anexternal charger, a wire/wireless data port, a card socket for a memorycard, a subscriber identification module (SIM) card, a user identitymodule (UIM) card, etc., an audio input/output (I/O) terminal, a videoI/O terminal, an earphone, etc. The interface unit 2800 may receive dataor power from such an external device, deliver the data or power to theelements included in the HMD device 2000, and transmit data stored inthe HMD device 2000 to an external device.

The power supply unit 2015 supplies power for operating the HMD device2000 to the elements of the HMD device 2000. Although not shown, thepower supply unit 2015 may include a battery for charging, and a cableor a cable port via which power may be received from the outside.

In general, the control unit 2600 controls overall operations of the HMDdevice 2000. For example, the control unit 2600 may execute programsstored in the memory 2400 to control the display unit 2030, the camera2050, the user input unit 2200, the output unit 2300, and thecommunication unit 2400.

The HMD device 2000 may be connected to the first device 1000, anddisplay an additional screen of a virtual image on a screen of the HMDdevice 2000 by receiving information regarding the additional screen ofthe virtual image from the first device 1000. According to an embodimentof the present disclosure, the HMD device 2000 may be connected to aserver (not shown).

The control unit 2600 may receive the information regarding theadditional screen of the virtual image and information regarding alocation on the display unit 2030 of the HMD device 2000 on which theadditional screen is to be displayed from the first device 1000 via thecommunication unit 2500.

According to an embodiment of the present disclosure, the control unit2000 may photograph the shape of the first device 1000 by using thecamera 2050, determine a location on which the additional screen of thevirtual image is to be displayed based on information regarding theshape of the first device 1000 and the information regarding theadditional screen, display the additional screen of the virtual imagenear a display screen on the first device 1000, and control theadditional screen of the virtual image to be linked to the displayscreen on the first device 1000.

In this case, the control unit 2600 may extract information regardingthe location and angle of a screen on the first device 1000 frominformation regarding the shape of the first device 1000, and determinea region in which the additional screen of the virtual image is to bedisplayed, based on the location and angle of a screen on the firstdevice 1000. For example, the control unit 2600 may determine thelocation and angle of the screen on the first device 1000 by receivinginformation regarding the shape of the first device 1000 from the firstdevice 1000, extracting feature points from the information regardingthe shape of the first device 1000, and comparing feature points of animage of the first device 1000 captured by the camera 2050 with thefeature points of the information regarding the shape of the firstdevice 1000.

Also, according to an embodiment of the present disclosure, the controlunit 2600 may control an additional screen of the HMD device 2000 linkedto a display screen on the first device 1000 according to a user inputperformed on the first device 1000 and a user input performed using thecamera 2050 of the HMD device 2000. Also, the control unit 2600 mayrecognize a user input on the additional screen through the camera 2050,ends the additional screen when the user input is recognized, andtransmit information indicating the end of the additional screen to thefirst device 1000.

The HMD device 2000 may display an additional screen on a screen thereofwithout being connected to the first device 1000.

In this case, the control unit 2600 photographs the first device 1000 inan actual space by using the camera 2050. The control unit 2600 mayphotograph the first device 1000 in the actual space in real time or apredetermined cycle.

Also, according to an embodiment of the present disclosure, the controlunit 2600 determines additional screens to be displayed around acaptured image of the first device 1000. In this case, the control unit2600 may display a user interface for selecting an additional screen ona screen of the HMD device 2000, and select a specific additionalscreen, based on a user input received via the user input unit 2200 orthe camera 2050.

Also, the control unit 2600 may determine a location in the vicinity ofthe first device 1000, on which an additional screen is to be displayed.

According to an embodiment of the present disclosure, since anadditional screen is displayed using the HMD device 2000, a limitedscreen of the first device 1000 may be expanded and a large amount ofvisual information and a convenient low-level interaction may beprovided.

An embodiment of the present disclosure may be embodied in the form of anon-transitory computer-readable recording medium having recordedthereon computer-executable instructions, e.g., a program module thatmay be executed using a computer. Examples of the non-transitorycomputer-readable recording medium may include not only an arbitraryavailable recording medium that is accessible by a computer but alsovolatile and non-volatile recording media, and separable/non-separablerecording media. Also, examples of the non-transitory computer-readablerecording medium may include computer storage media and communicationmedia. The computer storage media include the volatile and non-volatilerecording media and separable and non-separable recording media whichare manufactured according to an arbitrary method or technology to storeinformation such as computer-readable instructions, data structures,program modules or other data. The communication media generally includecomputer-readable instructions, data structures, program modules, otherdata of modulated data signals, other transmission mechanisms, andarbitrary information transfer media.

In the present disclosure, the term ‘unit’ may be understood asincluding a hardware component (such as a processor or a circuit) and/ora software component executed by a hardware component such as aprocessor.

It should be understood that various embodiments described herein shouldbe considered in a descriptive sense only and not for purposes oflimitation. Descriptions of feature points or aspects within eachembodiment of the present disclosure, should typically be considered asavailable for other similar feature points or aspects in other variousembodiments. For example, elements described as being included in asingle unit may be dispersed. Similarly, elements described as beingdispersed may be combined in a single unit.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

What is claimed is:
 1. A method of processing a screen on a device,performed by a head-mounted display (HMD), the method comprising:obtaining, by the HMD, an information related to an actual displayscreen displayed on the device; determining, by the HMD, an additionalscreen to be displayed based on the information related to the actualdisplay screen; displaying, by the HMD, the additional screen as avirtual image viewed near the actual display screen on the device. 2.The method of claim 1, further comprising detecting an input forselecting a content on the additional screen; in response to detectingthe input, displaying the additional screen corresponding to theselected content according to the detected input; and in response todetecting an input for selecting the additional screen, transmittingselection information corresponding to the additional screen to thedevice so that the device updates the actual display screencorresponding to the additional screen according to the selectioninformation received from the HMD and displays the updated actualdisplay screen on the device.
 3. The method of claim 1, wherein thescreen display request is generated by the device or the HMD accordingto a user input.
 4. The method of claim 1, wherein the informationrelated to the actual display screen comprises at least one of: shapeinformation of the device; a visual location of the additional screen tobe displayed visually near the actual display screen of the device; asize of the additional screen; content information related to theadditional screen; and state information of the device captured using acamera.
 5. The method of claim 4, wherein the state information of thedevice comprises a visual location and angle of a screen of the device.6. The method of claim 4, wherein the state information of the device isperiodically updated using the camera.
 7. The method of claim 1, whereinthe displaying of the additional screen comprises: capturing an image ofthe device; determining a visual location and direction of a screen ofthe device, based on the captured image of the device; and displayingthe additional screen on a predetermined visual location near the actualdisplay screen of the device, based on the visual location and directionof the screen of the device.
 8. The method of claim 1, wherein theadditional screen of the HMD is displayed as visually near the actualdisplay screen on the device by using the HMD device.
 9. The method ofclaim 7, wherein the determining of the visual location and direction ofthe screen of the device comprises: receiving shape information of thedevice from the device; extracting feature points from the shapeinformation of the device; and comparing feature points of the image ofthe device with the feature points of the shape information of thedevice to determine the visual location and direction of the actualdisplay screen of the device.
 10. The method of claim 1, furthercomprising controlling the actual display screen of the device and theadditional screen to be linked to each other according to user inputsperformed on at least one of the device or the HMD.
 11. The method ofclaim 10, wherein the controlling of the actual display screen of thedevice and the additional screen to be linked to each other comprisescontrolling the actual display screen on the device and the additionalscreen on the HMD to be linked to each other according to the userinputs performed on the respective the device or the HMD.
 12. The methodof claim 10, wherein the controlling of the actual display screen on thedevice and the additional screen to be linked to each other comprises:recognizing a command to end the additional screen according to a userinput received from the device; and ending the additional screen bytransmitting the command to end the additional screen to the HMD. 13.The method of claim 10, wherein the controlling of the actual displayscreen on the device and the additional screen to be linked to eachother comprises: recognizing a user input on the additional screen onthe HMD by using a camera; and ending the additional screen andtransmitting information indicating the end of the additional screen tothe device when the user input is recognized.
 14. A head-mounted display(HMD) apparatus for processing a screen by using a device, the HMDapparatus comprising: a display unit for displaying an image; acommunication unit configured to receive, via a network connection tothe device, an actual display screen on the device and informationrelated to the actual display screen displayed on the device; and acontrol unit configured to: obtain an additional screen based on theinformation related to the actual display screen obtained by thecommunication unit, display the additional screen on the display unit asa virtual image viewed near the actual display screen on the device,control the actual display screen on the device and the additionalscreen to be linked to each other according to a user input.
 15. Theapparatus of claim 14, wherein the control unit is further configuredto: detect an input for selecting a content on the additional screen, inresponse to detecting the input, display the additional screencorresponding to the selected content according to the detected input,and in response to detecting an input for selecting the additionalscreen, transmit selection information corresponding to the additionalscreen to the device so that the device updates the actual displayscreen corresponding to the additional screen according to the selectioninformation received from the HMD and displays the updated actualdisplay screen on the device.
 16. The apparatus of claim 14, wherein thecontrol unit is further configured to: photograph a shape of the deviceby using a camera, extract information regarding a visual location andangle of a screen of the device from state information based on theshape of the device, and determine a visual region in which theadditional screen is to be displayed, based on the visual location andangle of the screen of the device.
 17. The apparatus of claim 14,wherein the control unit is further configured to control the actualdisplay screen of the device and the additional screen according to atleast one of a user input performed on the device or a user inputrelated to the additional screen.
 18. The apparatus of claim 14, whereinthe control unit is further configured to: recognize a user input on theadditional screen through the camera, end the additional screen when theuser input instructs to end the additional screen, and transmitinformation informing the end of the additional screen to the device.19. A non-transitory computer-readable recording medium having recordedthereon at least one program including instructions for performing amethod of processing a screen on a device, performed by a head-mounteddisplay (HMD), the method comprising, the method comprising: obtaining,by the HMD, an information related to an actual display screen displayedon the device; determining, by the HMD, an additional screen to bedisplayed based on the information related to the actual display screen;displaying, by the HMD, the additional screen as a virtual image viewednear the actual display screen on the device.